A Fast Algorithm for Discovering Optimal String Patterns in Large Text Databases

We consider a data mining problem in a large collection of unstructured texts based on association rules over subwords of texts. A two-words association pattern is an expression such as (TATA, 30, AGGAGGT)) C that expresses a rule that if a text contains a subword TATA followed by another subword AGGAGGT with distance no more than 30 letters then a property C will hold with high probability. The optimized con dence pattern problem is to compute frequent patterns ( ; k; ) that optimize the con dence with respect to a given collection of texts. Although this problem is solved in polynomial time by a straightforward algorithm that enumerates all the possible patterns in time O(n 5 ), we focus on the development of more e cient algorithms that can be applied to large text databases. We present an algorithm that solves the optimized con dence pattern problem in time O(maxfk;mgn 2 ) and space O(kn), wherem and n are the number and the total length of classi cation examples, respectively, and k is a small constant around 30 50. This algorithm combines the su x tree data structure in combinatorial string matching and the orthogonal range query technique in computational geometry for fast computation. Furthermore for most random texts like DNA sequences, we show that a modi cation of the algorithm runs very e ciently in time O(kn log 3 n) and space O(kn). We also discuss some heuristics such as sampling and pruning as practical improvement. Then, we evaluate the e ciency and the performance of the algorithm with experiments on genetic sequences. A relationship with e cient Agnostic PAC-learning is also discussed.